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ABSTRACT 

This paper proposes a contextual evaluation method for 
computer-mediated teletraining systems. The proposed methods include: (1) 
quantification of results of the work on a generally recognized scale, in 
order to evaluate performance reached by the trainees following the 
achievement of the training task; (2) analysis of communication types used 
during each activity to resolve the task; and (3) questionnaires on the 
perception and usage of the different items of the environment that 
facilitate communication during the training session, in order to evaluate 
the satisfaction of actors with working in the mediated environment and with 
the proposed role. The techniques and tools of evaluation are described, 
including evaluation of the performance of the trainees, evaluation of the 
actors' satisfaction, and evaluation of the actors' behavior and system 
behavior; a table presents important factors and measurable or observed 
criteria in the evaluation of actors and system behavior. Considerations in 
the analysis of data and validation of results provided by the evaluation are 
also discussed. (MES) 
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here a contextual evaluation method The difficulty of this kind of 
^aluation comes fo>m the complexity of the real situation to which it apphes : the procedure 
bemg contextual, it is practiced on the situation, in real time and not in condition of toboratoiy 
where one observes some cntical factors by neglecting others and often the finality of tte 
tok. Con^ently, we have sought to understand the context of each computer mediated 
fa^g syste^ si^nom Methods that we propose are based on a quantification of results of 
Peifonnance reached by the, analysis of communication types 
actiwty to resolve the task in order to evaluate the behavior of actors and^ 
teh^r of the soci^techmcal system, and questionnaires on the perception and the usage of 
me dilterent items of the environment in order to evaluate the satisfaction of actors. 
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1. Introduction 



We propore here a contextual evaluation method This type of evaluation is carried out step bv sten bv 
following the evolution of ^h group of trainees under obLvation in the expS^tTfi^ew^ ffie 

J^XoiSfomft “ ’ evaluation is an experimental ^^tion because users are 

Pe diffic^ty ^ the contextud evaluation comes fixim the complexity of the real situation to which it applies* 
the procedure bemg contextual, it is practiced on the situation, in real time and not in condition of lalSratory 
wh^ one obseives some cnhcal fectors by neglecting others and often the finahty of the task Techniques S 
"'"Jit to be reaUy a^pted to the context and to objectives to reach. CWque^lTe reS to 

^^erifti^r SbsSlrtic^^ computer media^ training systems situation to seek there directly meaible 
characteristics (obs^ahons measures) or there mdirectly measurable characteristics (int^ews fine 

direussions, results ofthe work, measures). Methods that we piopore are based oT^ ^interviews, tree 

of the work on a generally recognized scale (for example, for the apprenticeship, notes 
“8=) i. onto to evaluate perfomtauce reached by the ttahteea foUowiug 

- a^ysrs Of commimcation types used during each activity to resolve the tasL In this aim, we suggest to 
conduct objectves observahons make by human observers and measures on the level and^ conS^th^ 

S^'S‘SutS\rsofw“ °^"I^ticeship (course, practical work and collaborative project. The 

- questionrwes on the p^ception and the usage of the different items of the environment that make easv the 
commumcation dunng the training session (for example, the artifects of telepresSL thTTuS^: 
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tepreseaadoi. of coocepK to loaro,....) in orier to ovaluoK the satisfectioo of artots ooth woridog in the 
mediated environment and with the proposed role. 

According to the specificity of each of these three sources of information, we have built tools to obtain necessary 
data for the evaluation. We present below this procedural approach. 

2 Techniques and Tools of Evaluation 

2.1 Evaluation of the Performances of the Trainees 

ThP nerformances reached by the trainees (training outcomes) foUowing the achievement of the peda^gical tek 
nrp ^^ated according to psycho-pedagogical evaluation methods. These methods are currently pracfaced m e 
Sg tninnr, «e orient e^Ination of the 

rfillah^ration^th cognitive technology properties used for the commumcatron m each type of session. The 
SSn^S'uTe teri fb, SnL'Si p^rical worint sesri™ and a aynchtonoua and connnnon, 

evaluation of the work for collective project sessions. 

The evaluation by test is traditional for the classical training and is based on the mea^ of t^ 
roanitive activities we can evaluate during training sessions. It concerns knowledge memorization, 

A' nA maotpTv levels The test is filled by each trainee after the chained training session course - 
^tical worL" . inSlt by the trainer and contains questions related to the three cognitive activities (Sandoz 

^fm^orization of information is linked to the perception of visual info^tion 

the textual or graphic form, hearing information transmit by oral way, by the tramer, durmg foe courw and visual 
and hearing infoi^tion (information presented in foe textual or graphic form and commOTted by ^ay). 

vSSlX SS to^ve identical problems (mainly in foe context of practical worl«)wh^ 
foe trainee has to^how that he has assimilated concepts by solvmg problems (a question or a problem 

pn*lem wrib tie «ali.y of «q»ired dumg to oow» 

^ works^ioo (O qdcriion ■ «w p^blem, tot torrire, a ™mg proce,, Ob to totaldy of 

knowledge). 

The pedagogical strategy of collaborative project aims to increase foe Imowl^e level of 
canabilitv to analyze a problem, to apply knowledge in order to solve a problem, to s>mfoesize an 
foe SSIss^^luti^ A^ulticriteria qualitative, quantitative and pedagogical evaluati^ « ^ 

ovalbario. .0 diffa»« riagea of to ^eo. ^“5 

succeedS^wifoout fault but with suggestions by foe tramer, succee^ but ^fo sn^ 
and graduated according to foe quality and foe independence of each group during foe project). 

2.2 Evaluation of the Actors Satisfaction 

The satisfaction of actors that interact with and via a computer system is intuitively correlated ^fo qu^ty 
If uSeTfoe There are elements or states of foe environment that stimulate or to mhibit^ 

oxperimtobl arid co»totol .pterion ^ 

discussions and questionnaire as efficient means to appreciate foe satisfoction, foe m^mty thus 

^Zrdato^theoretical aspects about foe content of these means neither on m^ods of data analysis thus 

obtained. The experimental evaluation offoe satisfaction is often an empincal approach. , 

cSmmnication with the other colleagues in foe group, satisfaction towards foe affluence and foe performance 
software tools, etc. ) to deepen its production mechamsm. 
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rr^L-~r“ 

the factonal analysis to determine dimensions of the satisfaction. He finds that the satisfection with 

These stupes show that the satisfection can be conceptualized and that questions for askine to users in tn 

Er?^E“~ 

SaSiiSvT^SSoS 2 we have to complete these questions in the situation of 

diff^ntiated quality of the media accortog to mleTL^I^Sroftte tl 

rtJl" ^ of coUea^es, audio to take a collective decision, etc. ), the mSc^Tty X'SiIw 
mediated ccZ^aSn/ simultaneously several modes of computer real time 

of eiqierimentation, if the mediated situation is completely new for the totality of the actors (they don’t 



SS’TL'Sot? cow^^acl wo* aod coUatoavo pioioot aosaioas a* descnbod m die amea. II 
presents the content of there questionnatres and points attached to each response according to the scale of Likert 
nlow ^ues mdicate that the user is satisfied with the situation and the vJS 

neutral viewpomt (or a normal situation) and raised values indicate the dissatisfection (see Coutaz & Balte 94^ 



23 Evaluation of the actors behavior and of the system behavior 

^e analysis of the communicative behavior and of the system performances during each session provides a lot 

Ch^iStldohJ'*- about the communication of the group in L mediated e^nment 

nS ao 2^ undertake this evaluation is a type « quick and dirty ethnography » method 

P^94], is^ of the ethno-methodology (Hughes et al. 1991) and motivated bjl^ie lackStJe and humans 
oW^’ of competence and views, we propose the following fectors as being relevant for 

objectve, of o„ evaluaOoo add we observe d.em i„ each session eontex. aorlrfing ,o^,SS Tab.l. 
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A/*frir<j anH Svstem Bchavior 


Factors 


(measurable or observed) Critena 


Individual Behavior 


Verbal behavior 
Not-verbal behavior 

Associative behavior resulted from 
personal motivation or re- 
organization of the planed work 


Density of dialogue 
Type of dialogue 
(Mentation in the working space 
Behavior in critical situations 
Information research 
Relationship with the trainer 
Relationships in the group 
(jiouping together in ordCT to achieve 
the task 


Group Behavior 


Length and sharing of work time 
Quality of the knowledge support 


Length of each task during the session 
Verbal interaction length 
Capability to plan work 
Difficulty of representation 
Difficulty of manipulation of the 
support 

Reliability of the software 


System Behavior 


Quality of the response of the 
system 

Quality of resources 


Periods of observability of actions on 
the interface 

Latency of reply of the system 
Latency of the network 
Input / Output Management 
Load of the CPU 

Load of the network ^ 



Table 1 : Important Factors and Criteria in the evaluation of actors and system behavior 



These methodological approaches of the experimental evaluation in our Rework imply that the evaluators use 
different techniques of quantitative and qualitative data coUection classified hereafter. 

each activity. 

sna^ofwork. research for help, coUective decision, co-operative execution 

Several fact*^ as quality of the equipment, of the operating ^stem and of the graphic manager, quality of th 
applications, quality of the network characterize the system behavior. 

Typical questionnaire interviews (quantitative and qualitative result^ 

2LiiS^ different levels of satisfaction or need in the context of each situation (see Rao 94).. 



3 Considerations on the analysis of data and the validation of results provided by the 
evaluation 



An other noint which is seldom Fecise in studies about the experimental evaluation is how to andyze of 
^talL^SvItio^. to which Z attach different significance and wb^h don’t have the same nature when the 
evaluation methodology of evaluation is based on a multiplicity of methods. 
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tendency ^ the research on methods of evaluation is the adaptation to the situation with the 

exploratoty ^ysis. Tbs orientation presents advantages and disadvantages beside the probability to obtain 
valid results (results which could be used faithfully in similar cases). proDanility to obtam 

methodolo^ of eval^tion ad^ted to the situation is the result of the reasoning of the evaluators from the 
context!^ situatoon and tte objectoves to reach. Therefore, it will depend a lot on infoimation and knowledge a 
Consequently, m a complex and new situation it is always desirable to use ^ral 

out tne ssnie tssk m the ssme enviro nm ent- ^ 

The experimental evaluation ^ the advantage of the multiplicity of methods (redundancy of the infoimation) 
appro^h (to understand increasingly complex situation). The r^undancy resulte of 
several solves of mformatoon on the same situation and nears us the truth on the reality. The bcremental 

Sons "" I “1' based on knowledge of specif 
“i? “?vidual. These observations have been built to underline the impoiS of 

S^t r evaluatoons and how to present these results. Techniques refer tria^ation of 

i " qt^totatove) and expressing different viewpoints (different roles of actors, 

)• This variety involves an interpretative 

Bpproach in the d^ta in order to synthesize results. ^ 

of A*!.!™® outeomes, it is necessary to interpret the cognitive level evaluation according to the 
ch^ten^c of *e tramee populatoon and to cogmtive properties of the media used in the training. 

of the satosfactoon it is necessary to interpret answers to questionnaires that represent personal 
actor pedagogical and technical viewpoints 

The analysis of consui^ted resources has to be made on different temporal scales according to whether one 
s'Wlier (machine, network which requires analysis, to the second or in und^^or the 

Of information. TTius, for training outcomes, we propose a statistical analysis 
distribution of Values) to compare these results with traiig 
^jectoves. If the mmbCT of tramees is smaU, two strategies can be adopted to increase the faith in results • to test 
two popi^toons with different level of initial knowledge but with the same general profile S ex^S? 

b ordCT to anb^ res^nses to questionnaires about the actors satisfection with the environment , we propose 
an ai^sis m different level. TTie analysis of unity of the tested population defines the correlation b^SSS 

“®“bers of the group). The aim of a discriminating analysis S identi^d 
— vOTfi^toon of the clusters (hypothesis test on equality of averagL clusters), the 
She combination construction of items - questions that summarize at 

pedagogical and technical quality of the experimental framework is 
^yzed from ^ks assigned to discnmmating and not discnfrninating items of the satisfaction 
For ai^ysis of a^ md ^stem behavior, we can interpret measures and observations to get a formative 
e^uatoon on the tehavior of ^ors and on the quality of the system (the reaction and the knowledge siqiport) in 

Se environment , and a summative evaluation on economic criS such 

as the time of ^rk and resources consumed for this work in order to the concrete limit of distance training 
sessions and to better understand where and how we can consider optimization. 

4 Conclasion 

to “btation is also up for evaluation». TTierefore we had 

evaluatoon means both for collecting data and for data analysis and 
achievements with training in CETTE environment (Giadinariu 
rhL?S^T^“ °° satosfectoon and on their opinion with this style of training, measurements to 

utilization, measurements and observations of on-going 
sessions (actor s behavior and external, unpredictable perturbations) ^ ^ 

f relationship that exists between the technical means quality and the 

1 ^ pedagogical that these means can svq^port. This duality is classic in the theory of the 

evaluaton of leaimng systems (Scriven 67) that anticipates as objective a quanhtotive evaluation (^“dZ^ 
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of the good usage of the technology) and formative-evaluative (analysis of means and pedagogical strategies 
when the both are in relationship of usage). 
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